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This article examined a unique collaborative initiative between a team of graduate level faculty to 
improve the quality of course development and delivery using a collaborative review process. The 
collaborative review of teaching has become a widely accepted means for assuring quality distance 
learning coursework and for the purpose of evaluating the effectiveness of online teaching. The 
criteria of the Quality Online Learning and Teaching (QOLT) Assessment for Peer Reviewers 
instrument were used to measure the quality indicators that address good teaching and learning 
practices. The authors outline the steps in the process of developing and managing such an online 
graduate distance learning program. 


O nline instruction has significantly 
impacted graduate education 
programs. Online learning 
opportunities are expanding rapidly in the 
university setting to meet the changing needs 
of the higher education student. Agreeing that 
online education is critical to their long-term 
strategy, postsecondary institutions in United 
States report a total of 7.1 million students 
taking at least one online course in fall 2012 
(Allen & Seaman, 2014). Experiencing 
yearly growth, the percent of online 
enrollment as a percent of the total 
enrollment in higher education grew from 
9.6% in 2002 to 33.5% in 2012 (Allen & 
Seaman, 2014). Although there has been a 
substantial growth in the delivery of 
academic programs and courses by distance 
learning provided by institutions of higher 
education, the need for quality learning must 
be realized. Building faculty expertise and 
skill in this endeavor are desired and 
considered essential. Faculty who are 
teaching in the College of Education and 
Behavioral Science (COEBS) in this study 


have been and continue to be pioneers of 
distance learning for graduate education 
majors. 

The growing demand for distance 
education courses is occurring at the same 
time that higher education in the United 
States is evolving. External pressures are 
converging to challenge the traditional mode 
of providing higher education, particularly in 
public institutions. Decreased state funding, 
escalating tuition, a depressed economy, 
advances in technology and techno-literacy, 
and students’ expectations of convenience 
and flexibility are major external drivers of 
change. Simultaneously, there are internal 
pressures to drastically reduce costs and 
increase revenues in public universities and 
colleges. These external and internal 
pressures combine to influence change in the 
traditional modes of teaching students. 
Distance education programs can be a highly 
successful approach to meet the demands of 
an escalating number of students while also 
reducing costs and generating revenue to 



provide quality education (Kuruvilla, Norton, 
Chalasani & Gee, 2012). 

The perceived flexibility associated 
with online learning and satisfaction in 
distance education classes is a major factor 
for students choosing to enroll in e-leaming 
courses. Students are more likely to enroll if 
they can benefit from both the time and place 
flexibility that the distance education 
medium offers (Arbaugh & Duray, 2002). 
The millions of students enrolled in online 
courses today provide clear evidence that this 
modality is meeting market demand for 
students. While online education broadens 
the diversification of educational 
opportunities, it also creates a new paradigm 
for online teaching and development in 
higher education. 

Background of Online Program in 
Education 

Arkansas is experiencing substantial 
growth in the delivery of academic programs 
and courses by distance learning provided by 
institutions of higher education, located both 
within and outside of the state. There is 
increasing demand by students in Arkansas 
for academic courses and programs offered 
by distance learning and for some students or 
prospective students (such as those with 
handicapping conditions, work and parental 
obligations, economic challenges, and other 
constraints) it is the only way in which they 
may pursue their education (Chancellor’s 
First Friday, Arkansas State University, 
2012 ). 

This university identified in 2008 the 
need to begin offering complete high-quality 
academic programs in selected areas online. 
The courses offered in these programs are 
comprised of intensive seven-week time 
blocks with one to two weeks between 
courses. Students may enter this 30 
semester-credit-hour-graduate program at the 
onset of any seven-week course (except for 
the final capstone course, which must be 


taken last), and may conclude successfully in 
approximately 18-24 months (Chancellor’s 
First Friday, Arkansas State University, 
2012 ). 

In review of essential attributes of 
distance learning, the development of 
pedagogy and a valid course curriculum were 
viewed as having utmost importance. 
Developing a course for online instruction 
requires content knowledge and 
understanding of the interactivity, 
technological requirements, and best 
practices in the asynchronous environment. 
The goal is to ensure academic rigor in these 
classes, designing graduate courses taught 
online as close to traditional classes as 
possible. It has been suggested that the social 
phenomenon of community may be put to 
good use in the support of online learning. 
Deemed as a means to increase collegial 
support and improvement of teaching, faculty 
collaboration demonstrates an approach for 
designing online courses based on best 
practices (Carney, Dolan & Seagle, 2015). 

Collaborative Review Process 

As the need for online instruction 
continues to grow, the need for quality 
control of the course work to maintain course 
quality and credibility becomes a 
fundamental element of program delivery. 
Establishing a quality review process for 
online courses has therefore become critical. 
A fonnal review of online courses can 
measure the quality of the course and reveal 
changes needed for improvement in the 
application of the technology, the 
pedagogical processes, and overall clarity in 
the presentation of a course. Once program 
courses were developed and offered online, 
the college deemed it necessary to make sure 
all courses maintained their authenticity and 
level of rigor. The process of establishing and 
conducting a collaborative review based on a 
proposed framework or checklist is essential 
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for examining all aspects of the quality of an 
online course. 

Collaborative review is not a new 
pedagogical approach however, once aligned 
with online course standards, it can offer 
tools for quality assurance processes. Peer 
feedback refers to reciprocal teaching in 
which paired teachers provide assistance to 
observe one another as they incorporate new 
teaching techniques in the classroom 
(Vidmar, 2005). The primary goal of peer 
feedback is to promote professional 
development, collaboration, and self- 
assessment (Vidmar, 2005; McTighe & 
Emberger, 2006). The process involves 
giving and receiving feedback in both written 
and verbal format. A formal review of online 
courses can measure the quality of the course 
and reveal changes needed for improvement 
in the application of the technology, the 
pedagogical processes, and overall clarity in 
the presentation of a course. Faculty 
acknowledged the critical need for 
developing standards that assure quality of 
their online courses. The process of 
establishing and conducting a quality review 
based on a proposed framework or checklist 
is vital in examining aspects of best practices 
for the essential attributes of distance 
learning. 

A five-phase process was developed 
to ensure objectivity and credibility of 
Quality Matters in improving online teaching 
and learning. The steps in the collaborative 
review process are iterative, in order to 
address issues and challenges encountered at 
each stage, thus refining the process. As 
shown in Figure 1, the chart illustrates the 
collaborative review process. 

Figure 1 . The Collaborative Review Process 



The college selected the Southern 
Region Education Board (SREB) Quality 
Matters Rubric Standards, “Quality Online 
Learning & Teaching (QOLT) Assessment 
for Peer Reviewers,” to review the quality of 
the online program instructional plan and 
design. The process provided the opportunity 
for fundamental ownership in the outcome 
and level of synergy that ultimately ended in 
a better designed and more useful tool. 
Initially, the review begins with a self-review 
whereas the professors complete a thorough 
review of their course after course documents, 
assignments, and assessments were updated 
for the upcoming semester. Upon completion 
of the self-review the professor would then 
select another professor from a different 
content area. The peer review process begins 
when the professor reviews the course for 
cohesiveness and provides feedback using 
the Online Course Quality Checklist (see 
Appendix for Table 1). During the 
collaborative review process, participating 
faculty engaged in a valuable team-based 
learning experience. The diversity among the 
faculty allowed experts from various fields to 
critique student and faculty teaching and 
learning. Next, the department chair would 
review the Online Quality Checklist 
completed by the professors and peer 
collaborators to ensure that each indicator has 
been addressed during the review. Following 
the peer review, the department chair debriefs 
with all faculty to provide a complete review 
of each course involved in the peer review 
process. After the review, the department 
would then devise a collaborative plan that 
includes strengths, areas of need, and 
recommendations for changes. 

When the first three-stages of the 
review process is completed (self, peer 
collaborator, and chair review) and issues 
have been determined for remediation, the 
course professor has to determine how to 
remediate the matter and make the necessary 
revisions to complete the review cycle. 
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Fundamental to the success of the 
collaborative review process is that each 
professor has a mutual respect for the peer’s 
expertise. Furthermore, the course professor 
considers what his/her peer collaborator and 
the chair reviews have determined about 
what needs to be added, taken out, or changed 
in the course. Once the professor of the 
course makes the appropriate changes within 
the online course he/she will then compile a 
list of the changes on the course revision 
section of the Online Quality Course 
Checklist and return the form back to the 
department chair to complete the file on that 
course. 

After all changes were made by the 
professor the Interactive Teaching & 
Technology Center (ITTC) faculty support 
coordinator reviews the course in terms of 
course design and accessibility in preparation 
for online course release. Upon successful 
completion of all reviews by the professor, 
peer collaborator, department chair, and 
ITTC faculty support coordinator, the course 
is ready for full release to students for the 
upcoming tenn. 

To further enhance the collaborative 
process the assessment office begins the 
process of refining or developing assessment 
plans for continuous improvement. The 
“continuous improvement” process includes 
outcome development, measure selection, 
cyclical assessment planning, benchmark 
designation, and accountability structures. 
These components ensure that the programs 
adhere to regional accreditation standards, 
specialized accreditation standards, and state 
mandates. The final phase of the 
collaboration process defines how well 
students are learning within academic 
programs, it also informs action plans for 
future improvement and increased student 
learning in all academic programs, regardless 
of the modality. 


Barriers and Limitations of Peer 
Review. Peer review is one of the most 
valuable collaborative processes employed; 
nonetheless, it does not come without its 
challenges. Throughout the peer review 
process, a few issues arose that impeded 
progress. The decision as to which rubric 
would be employed was the first source of 
contention. It was difficult to get faculty buy- 
in for the use of the Quality Matters rubric 
initially, simply because faculty were not 
sure that this rubric embodied everything 
they felt needed to be evaluated. Some 
faculty submitted other rubrics for 
consideration, but it was the Quality Matter 
rubric that emerged as the best one for peer 
review of online courses within our college. 

Another issue was the matter of 
interpreting the rubric. It is important that 
rubric scorers interpret the rubric in exactly 
the same way to ensure that the scores are 
valid. To this end, inter-rater reliability was 
established to ensure agreement among raters. 
When inter-rater reliability was not high 
enough to constitute agreement among raters, 
the discrepancies were discussed and raters 
were able to reach consensus on the rubric 
items in question. Another issue to arise was 
that of subjectivity. While rubrics have the 
inherent ability to diminish subjectivity, this 
is not always a guarantee. Our experience 
was such that subjectivity was the root of 
much disagreement between raters. To 
combat this issue, raters had extensive 
conversations and training on the rubrics in 
an effort to maintain objectivity. In some 
cases, an additional peer reviewer was 
brought in to eliminate disparities among 
reviewers. 

Furthermore, there were logistical 
issues that had to be considered. It is 
imperative that multiple raters had access to 
the course shell at any given time so that the 
reviews could be completed as efficiently as 
possible. The challenge with granting access 
to the author of the course and multiple 
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reviewers is that of privacy. While the 
majority of faculty members welcomed the 
collaborative process, some faculty did 
express concerns regarding academic 
freedom of course development. In these 
cases, the professor’s evaluation and peer’s 
evaluation of the course occurred 
simultaneously. This gave both reviewers the 
opportunity to engage in open dialogue, 
eliminating anxieties about the infringement 
of knowledge and expertise. 

Implications for Practice 

The introduction of online teaching 
and development requires new pedagogical 
and technological skills (Garrison, 2011). 
Developing these skills often requires 
advanced knowledge of instructional design 
aimed at increasing online learning. 
Collaborative reviews in distance education 
demonstrate the value of professional 
learning. Faculty reviewers have responded 
favorably to the collaborative review process 
and feel they are able to accurately assess one 
another’s work. The practice of 
implementing such a review process can 
provide a useful model for the design and 
implementation of online course 
development. Conversely, faculty members 
were able to construct new meanings through 
collaboration and the integration of multiple 
perspectives through interaction and 
negotiation with each other (Hung, Tan & 
Koh, 2006). Although collaborative work can 
be somewhat challenging in any situation, the 
value of the activity can be more than just the 
learning outcome. It is the sense of learning 
how to use the technology together that 
allowed graduate professors of education to 
become knowledge constructors (Jonassen, 
2000), actively constructing their own 
knowledge through faculty collaboration. 

Indeed, Arbaugh (2008) affirms that 
higher-order learning experiences are 
obtained through a community of inquiry 


composed of faculty to achieve the desired 
outcome. With a better understanding about 
collaborative reviews for online teaching, 
how they work, and how they develop/evolve, 
this new learning paradigm will guide faculty 
in the use of pedagogically infonned models 
to enhance online course development and 
design. Damoense (2003) further suggested 
that learning that focuses on collaboration 
between the instructor and learners may 
ultimately increase interactivity and provide 
an authentic environment for learning. The 
authors note that because online learning and 
teaching is still evolving, there is more to be 
studied. Additionally, more well-designed 
research is needed on effective quality 
assurance and measuring instructional 
effectiveness in online courses. 

Conclusion 

As technology continues to evolve 
efforts to further the possibilities for relevant 
and interactive distance learning courses 
steadily increase. Most likely, online learning 
and teaching opportunities will continue to 
grow, and this should create new and 
innovative pedagogy and technology that 
support online teaching and development. To 
ensure the relevance and rigor of online 
learning, it is important that programs 
develop an effective assessment process. 
This collaborative review process was overall 
well received by the faculty teaching online 
graduate education courses. The process 
accomplished the mission of improving 
teaching effectiveness as well as improving 
the college’s quality of online programs. 
While the evidence is anecdotal, the faculty 
within the college participating in this study 
describes positive collaborative learning 
experiences. It was determined that with 
adequate planning and technical support, and 
continuous program monitoring, distance 
learning can be a practical opportunity for 
preparing professionals in the field of 
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education and leadership. The collaborative 
review process is noted as a best practice in 
education and has proven to be a strong 
indicator of success in the development of 
online learning. 
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Appendix 


Table 1 

Online Course Quality Checklist 

Course Title: _ 

Course Professor: _ 

Semester: _ 

Year: _ 

Professor Evaluation of Course: _ 

Peer Evaluation of Course: _ 

Comment: 

Course Overview and Introduction 

1.1 Instructions make clear how to get started and where to find various course components. 

1.2 Netiquette expectations for are stated clearly. 

1.3 The professor posts a welcome letter in which he/she welcomes students to the class, introduces him/herself, and 
explains the purpose of the course. 

1.4 Students are asked to introduce themselves to the class via a week-one discussion board. 

1.5 Course-specific proficiencies, beyond those which have been identified as being essential to participation in the 
program, regarding technical skills expected of the student are clearly stated. 

1.6 A syllabus using the approved College of Education & Behavioral Science CAEP format is provided to students. 

1.7 The grade book is updated in regard to due dates for assignments, designations for inclusion and exclusion of 
assignments for grade calculation purposes, and point values of each assignment. 

1.8 A link to the student survey regarding evaluation of the course is embedded as an activity in week 7. 

Learning Objectives 

2.1 The course learning objectives describe outcomes that are measurable. 

2.2 Learning objectives are stated clearly and written from the perspective as to what candidates will know and be able 
to perform. 

2.3 Learning outcomes from the content area standards are included among the learning objectives. 

Assessment and Measurement 

3.1 The course grading policy is stated clearly. 

3.2 Specific, clear instructions and descriptive criteria are provided for the evaluation of students’ work and 
participation. 

3.3 Rubrics are clear, directive and conform to the unit format (i.e. Exceptional, Adequate, Inadequate scale). 

3.4 The assessments selected measure the stated learning objectives and are consistent with course activities and 
resources. 

3.5 The assessment methods that are employed are varied and appropriate to the content being assessed. 

3.6 “Self-check” or practice assignments that are utilized provide timely and useful feedback to students. 

3.7 “Real-world,” authentic learning experiences require students to apply course concepts and objectives.* 

3.8 Remediation activities provide students at least a second chance to meet course outcomes. 

Resources and Materials 

4.1 A variety of instructional materials contribute to the achievement of the stated course and module/unit learning 
objectives. 

4.2 All resources and materials used in the course are appropriately cited. 

Learner Engagement 

5.1 Opportunities for student-to-student interactions/communication exist to promote learning and networking.* 

5.2 The course design affords opportunities for student-professor interactions that enhance learning. 

5.3 The course has at least one activity in which students engage with other professionals in field-based work.* 

5.4 Students engage in a variety of learning activities that address a variety of learning styles.* 

5.5 Course activities engage students in critical reasoning, problem solving, and higher-order thinking such as 
application, synthesis, and evaluation.* 

Learner Support 

6.1 Navigation throughout the online components of the course is logical, consistent, and efficient. 

6.2 Instructions on how to access resources at a distance are sufficient and easy to understand and follow. 

6.3 Course instructions answer, if necessary, basic questions related to course-specific research, writing, technology, 
etc., and/or link to tutorials and/or other resources that provide the information. 

6.4 Professors post and keep virtual office hours to enhance assistance to and interaction with students. 

Accessibility 

7.1 The course incorporates ADA standards and reflect conformance with institutional policy regarding accessibility in 
online courses. 

7.2 Course pages and course materials provide equivalent alternatives to auditory and visual content. 

Course Revision 

8.1 Enhancements of the course are continuous and are based, in part, on data from student and peer feedback. 

8.2 The course architecture permits the professor to add and/or edit content and assessments.* 

Additional Comments and Recommendations: 


All items denoted by * are from SREB’s November, 2006 “Checklist for Evaluating Online Courses.” All other items are from “Quality Matters 
Rubric Standards,” 2008-2010 edition. 
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